QUESTIONNAIRE SINA ASHRAFI
SINA ASHRAFI ' ' ' ' The QUESTIONNAIRE Although the term 'questionnaire' is one that most of us are familiar with, it is not so straightforward a task to provide a precise definition for it. The term is partly a misnomer because many questionnaires do not ''contain any, or many, real questions that end with a question mark. Indeed, questionnaires are often referred to under different names, such as 'inventories, ' 'forms, ' 'opinnionaires, ' 'tests, ' 'batteries, ' 'checklists, ' 'scales, ' 'surveys, ' 'schedules, ' 'studies, ' 'profiles, ' 'indexes/indicators, ' or even simply 'sheets' (Aiken, 1997). Second, the general rubric of 'questionnaire' has been used by researchers in at least two broad senses: (a) ''Interview schedules, ''like the ones used in opinion polls, when someone actually conducts a live interview with the respondent, reading out a set of fixed questions and marking the respondent's answers on an answer sheet. (b) ''Self-administered pencil-and-paper questionnaires, ''like the 'consumer surveys' that we often find in our mail box or the short forms we are asked to fill in when, for example, checking out of a hotel to evaluate the services. '''What do questionnaires measure?' Broadly speaking, questionnaires can yield three types of data about the respondent: factual, behavioral, andattitudinal. 1. Factual questions ''(also called 'classification' questions or 'subject descriptors') are used to find out about who the respondents are. They typically cover demographic characteristics (e. g., age, gender, and race), residential location, marital and socioeconomic status, level of education, religion, occupation, as well as any other background information that may be relevant to interpreting the findings of the survey. Such additional data in L2 studies often include facts about the learners' language learning history, amount of time spent in an L2 environment, level of parents' L2 proficiency, or the L2 coursebook used. 2. ''Behavioral questions ''are used to find out what the respondents are doing or have done in the past. They typically ask about people's actions, life-styles, habits, and personal history. Perhaps the most well-known questions of this type in L2 studies are the items in language learning strategy inventories that ask about the frequency one has used a particular strategy in the past. 3. ''Attitudinal questions ''are used to find out what people think. This is a broad category that concerns ''attitudes, opinions, beliefs, interests, and values. ''These five interrelated terms are not always distinguished or defined very clearly in the literature. • ''Attitudes ''concern evaluative responses to a particular target (e. g., people, institution, situation). They are deeply embedded in the human mind, and are very often not the product of rational deliberation of facts - they can be rooted back in our past or modeled by certain significant people around us. For this reason, they are rather pervasive and resistant to change. • ''Opinions ''are just as subjective as attitudes, but they are perceived as being more factually based and more changeable. People are always aware of their opinions but they may not be fully conscious of their attitudes (Aiken, 1996). • ''Beliefs ''have a stronger factual support than opinions and often concern the question as to whether something is true, false, or 'right'. • ''Interests ''are preferences for particular activities. • ''Values ''on the one hand concern preferences for 'life goals' and 'ways of life' (e. g., Christian values); on the other hand they are also used to describe the utility, importance, or worth attached to particular activities, concepts, or objects (e. g., instrumental/ utilitarian value of L2 proficiency). '''Advantages' The main attraction of questionnaires is their unprecedented efficiency in terms of (a) researcher time, (b) researcher effort, and © financial resources. By administering a questionnaire to a group of people, one can collect a huge amount of information in less than an hour, and the personal investment required will be a fraction of what would have been needed for, say, interviewing the same number of people. Furthermore, if the questionnaire is well constructed, processing the data can also be fast and relatively straightforward, especially by using some modern computer software. These costbenefit considerations are very important, particularly for all those who are doing research in addition to having a full-time job (Gillham, 2000). Disadvantages Although the previous description of the virtues of questionnaires might suggest that they are perfect research instruments, this is not quite so. Questionnaires have some serious limitations, and some of these have led certain researchers to claim that questionnaire data are not reliable or valid. I do not agree with this claim in general, but there is no doubt that it is very easy to produce unreliable and invalid data by means of ill-constructed questionnaires. In fact, as Gillham (2000, p. 1) points out, in research methodology "no single method has been so much abused. " Simplicity and superficiality of answers Because respondents are left to their own devices when filling in selfcompleted questionnaires, the questions need to be sufficiently simple and straightforward to be understood by everybody. Thus, this method is unsuitable for probing deeply into an issue (Moser & Kalton, 1971) and it results in rather superficial data. The necessary simplicity of the questions is further augmented by the fact that the amount of time respondents are usually willing to spend working on a questionnaire is rather short, which again limits the depth of the investigation. '' '' Unreliable and unmotivated respondents Most people are not very thorough in a research sense, and this is all the more true about dealing with questionnaires - an activity which typically they do not enjoy or benefit from in any way. Thus, the resuits may vary greatly from one individual to another, depending on the time and care they choose or are able to give (Hopkins, Stanley, & Hopkins, 1990). Respondents are also prone to leave out some questions, either by mistake or because they did not like them, and LowB (1999) presents empirical evidence that respondents also often simply misread or misinterpret questions (which of course renders the answers false). If returning the questionnaires to the survey administrator is left to the respondents (for example in a mail survey), they very often fail to do so, even when they have completed it. In such 'distant' modes, the majority of the respondents may not even bother to have a go at the questionnaire. After all, don't we all think, from time to time, that the questionnaires we receive are an absolute nuisance... ? '' '' Respondent literacy problems Questionnaire research makes the inherent assumption that the respondents can read and write well. Even in the industrialized world this is not necessarily the case with regard to the whole population: Statistics of about 5%-7% are regularly quoted when estimating the proportion of people who have difficulty reading, and the number of those who are uncomfortable with writing is even bigger. It is therefore understandable that for respondents with literacy problems, filling in a questionnaire can appear an intimidating or overwhelming task. '' '' Little or no opportunity to correct the respondents' mistakes Questionnaire items focus on information which the respondents know best, and therefore the researcher has little opportunity to double- check the validity of the answers. Sometimes respondents deviate from the truth intentionally (see further) but it is also common that - as just mentioned - they simply misunderstand or forget something, or do not remember it correctly. Another fairly common situation is when informants do not know the exact response to a question yet answer it without indicating their lack of knowledge. Without any personal contact between the researcher and the informant, little can be done to check the seriousness of the answers and to correct the erroneous responses. '' '' Social desirability (or prestige) bias The final big problem with regard to questionnaires is that people do not always provide true answers about themselves; that is, the results represent what the respondents report ''to feel or believe, rather than what they ''actually ''feel or believe. There are several possible reasons for this, and the most salient one is what is usually termed the ''social desirability ''or ''prestige bias. ''Questionnaire items are often 'transparent, ' that is, respondents can have a fairly good guess about what the desirable/acceptable/expected answer is, and some of them will provide this response even if it is not true. The most extreme example of a 'transparent' question I have come across was in the official U. S. visa application form (OF 156): "Have you ever participated in genocide?" Although most questionnaire items are more subtle than this, trying to present ourselves in a good light is a natural human tendency, and this is very bad news for the survey researcher: ''Self-deception Self-deception ''is related to social desirability but in this case respondents do not deviate from the truth consciously but rather because they also deceive themselves (and not just the researcher). As Hopkins et al. (1990, p. 312) point out, human defense mechanisms "cushion failures, minimize faults, and maximize virtues so that we maintain a sense of personal worth. " People with personality problems might simply be unable to give an accurate self-description, but the problem of self-delusion may be present on a more general scale, though to a lesser degree, affecting many other people. ''Acquiescence bias Another common threat inherent to self-completed questionnaires is acquiescence, ''which refers to the tendency for people to agree with sentences when they are unsure or ambivalent. Acquiescent people include "yeasayers, " who are ready to go along with "anything that sounds good" (Robinson, Shaver, & Wrightsman, 1991, p. 8), and the term also covers those who are reluctant to look at the negative side of any issue and are unwilling to provide strong negative responses. ''Halo effect The halo effect ''concerns the human tendency to overgeneralize. If our overall impression of a person or a topic is positive, we may be disinclined to say anything less than positive about them even if it comes to specific details. For many students, for example, a teacher they love is 'perfect' in everything he/she does - which is obviously not true. And similarly, if we do not like someone, we - quite unfairly - tend to underestimate all his/her characteristics. resulting bias poses a serious threat to the validity of the data. We should note that this threat is not necessarily confined to 'subjective' attitudinal items only. As Oppenheim (1992) warns us, even factual questions are often loaded with prestige: people might claim that they read more than they do, bathe more often than is true, spend more time with their children, or give more to charity than actually happens, etc. In general, questions concerning age, race, income, state of health, marital status, educational background, sporting achievements, social standing, criminal behavior, sexual activity, and bad habits such as smoking or drinking, are all vulnerable (Newell, 1993; Wilson &McClean, 1994). ''Fatigue effects Finally, if a questionnaire is too long or monotonous, respondents may begin to respond inaccurately as a result of tiredness or boredom. This effect is called ''the fatigue effect, ''and it is obviously more likely to influence responses toward the end of the questionnaire.